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ABSTBACT 

Beform in science and mathematics has moved from rote 
learning of facts and computation skills toward the learning of a 
structure of a subject matter. At present there is little empirical 
evidence to support any contentions that there is a match between the 
subject-structure taught and the cognition in subjects* memories 
resulting from the instruction. This paper presents methods for 
examining subject-matter structure in prose material and methods for 
examining representation of a subject-matter structure in subjects* 
memories. Data bearing on the validity of structural interpretations 
of these measures are examined. Such measures provide methods which 
might be used to evaluate the extent to which a mathematics 
curriculum communicates the structure it has been developed to 
communicate. (JP) 
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SOMH METHODS FOR EXAMINING 
CONTENT STRUCTURE AND COGNITIVE STRUCTURE 
IN MATHEMATICS INSTRUCTION^ 
Richard J. Shavelson 
University of California at Los Angeles 

Refonn in, science and mathematics curricula has moved from rote 
learning of facts and computation skills toward the learning of a struc- 
ture of a subject matter. The reasons, commonly cited for this shift, 
are that structural knowledge: (a) is required for a full understanding 
of the subject matter, (b) leads to solutions of problems not otherwise 
solvable as Branca (1974) pointed out, (c) leads to an aptitude for 
learning, and (d) results in intellectual excitement. These reasons for 
teaching structure are admirable and certainly to be valued, but there 
is little empirical evidence to support them (for exceptions, see 
Shnvelson, in press). 

An initiaT step in examining the claims for learning structure is 
to represent, in a clear and objective way, the structure in the to-be^- 
learned subject matter. But a problem arises immediately since a subject- 
matter structure can be represented in a variety of ways. One approach 
is to arrange a subject-matter structure hierarchically; the exact 
arrangement depends upon the psychological model underlying the analysis 
(cf* Ausubel , 1963; Bruner, 1965; Gagne, 1962). A second approach is 
content analysis (Berelson, 1954). The universe of content is defined, 
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a categoi^y system is devclopod to partition this universe, a unit of 
analysis is selected, and a measurement is taken, 

A third approach is to use a graph structure to represent concepts 
(broadly defined) and their interrelations in a subject matter (Crothers, 
1972; Frederiksen, 1972; Gee^lin, 1973; Shavelson, 1970, 1972)- The 
points or nodes on the graph represent concepts, and the lines represent 
interrelations 'between concepts as specified by the syntactic and/or 
semantic characteristics of (say) the prose materials used to communicate 
the subject-matter structure. 

Schwab (1964) has aptly criticized the hierarchical approach of 
mapping structure with an underlying psychological model as changing the 
subject-matter structure itself. Content analysis is too restrictive an 
approach since it focuses more on, say, words than on their interrela- 
tions. Any notion of structure rests at least as much on these inter- 
relations as on the words themselves. The third approach, representing 
a subject-matter structure es the interrelations among concepts in the 
subject matter, seems to correspond most closely with what curriculum 
specialists consider to be subject-matter structure. 

Once a method for representing subject-matter structure has been 
selected, a second problem arises; that of examining the representation 
of subject-matter structure in the memories of students. Measurement 
techniques should produce data which are consistent with a specified 
definition of structure and which can be compared with data representing 
subject-matter structure. Three such methods are word association, card 
sorting* and graph building. 

The purposes of this paper are to: (a) present a method for examin- 
ing subject-matter structure in prose materials (b) present methods for 
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for examining representations of a subject-matter structure in subjects* 
memories, and (c) present data bearing on the validity of structure 
interpretations of data obtained from these methods. 



'^tructure: Some Definitions 
Structure refers to an assemblage of identifiable elements and the 
relationships between those elements. Structure may be objective and 
real or subjective and internal (Shavelson, 1972). The structure in 
instructional materials is referred to as content structure : the web of 
concepts (words, symbols) and their interrelations in a body of instruc- 
tional material. This definition is similar to Schwab's (1962); a 
discipline is defined as a bundle of facts and the interrelations 
between those facts. Finally, structure in the student*s memory is 
referred to as cognitive structure : a hypothetical construct referring 
to the organization (relationships) of concepts in memory. 



Method for Examining Content Structure 
Content structure may be examined by means of the theory of direc- 
ted graphs (or digraph theory),^ 

This. . .theory is concerned with patterns of relation- 
ships among pairs of abstract elements. As such, 
digraph theory makes no reference to the empirical 
world. Nevertheless, it has potential usefulness to 
the empirical scientist, for it can serve as a mathe- 
matical model of the structural properties of any 
empirical system consisting of relationships among 
pairs of elements (Harary, Norman, & Cartwright, 1965, 
p. 9). 

If an appropriate coordination is made so that each 
entity of an empirical system is identified with a 
point and each relationship about structural proper- 
ties of the obtained digraph there are corresponding 
true statements about structural properties of the 
empirical system (Harary et al., 1965, p. 22). 



As an example of tfr^ application of digraph theory to the examina- 
tion of subject-matter structure in prose material, consider the subject 
matter of operational systems. As Branca (1974) pointed out^ an opera- 
tional system is defined as a set together with a binary operation on 
the set. An operational system may possess any or none of the following 
properties: associativity, commutativi ty, identity element, and round- 
ness. When digraphs are used to map structure in instructional material, 
words denoting concepts in operational systems are represented as points 
on the digraph and the links between concepts, .as specified by syntac- 
tical and semantic characteristics of the prose, as directed lines on 
the graph. The following steps are taken in mapping instructional 
material onto a digraph: 

1. Identify the key concepts in the subject matter. In opera- 
tional systems, the following concepts v/ere identified by the curriculum 
developers: Associativity, Binary Opev^ation, Commutativi ty. Element, 
Finite/Infinite, Fundamental Properties, Identity Element, Inverses, 
Ordered Pair, Operational System, Roundness, and Set. 

2. If two or more -key concepts are found in a sentence, the 
sentence is diagramed using the procedure due to Warriner and Griffiths 
(1957). For example, the sentence, "A set is a collection of elements," 
was diagrammed as: 



3. Using conversion rules for transforming a sentence diagram 
into a digraph (Shavelson, 1970; Shavelson & Geeslin, 1973), a digraph 
is obtained for each sentence. In the example, the following digraph 
was obtained: 




set .j is [ collection 




The symmetric relation between Set and Collection is specified by the 
rule for linking verbs: "a linking verb does not specify action and is 
to be digraphed as a symmetric relation between two points,^' The 
symmetric relation between Collection and Element is specified by the 
rule for prepositions: '*if the preposition does not specify direction, 
the relation is digraphed symmetrically/' 

4. Each sentence digraph, then, is converted into a distance 
matrix for all digraphs combined. The following procedure is used. The 
distance between two points on a digraph is the number of lines in the 
shortest path connecting the two points. In the digraph above, the 
distance betv;een Set and Connection is 1, and the distance between Set 
and Element is 2. Those sentence digraphs were combined into one 
digraph representing content structure. From this overall digraph a 
distance matrix was formed using an algorithm given by Harary et al. 
(1965, pp. 135-136). 

5. The data in the distance matrix were considered dissimilari- 
ties and examined with Johnson's (1967) hierarchical clustering technique. 

Methods for Examining Cognitive Structure 
Cognitive structure was examined by the methods of word association, 
card sorting, and graph building. With these methods, the order of 
response retrieval or the categorization of concepts was assumed to 
reflect at least a significant part of cognitive structure. For each 
test, the 12 key concepts were presented to two persons who had developed 
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the operational systems curriculum used in the digraph analysis (and in 
other studies). 

For the word-association test, the curriculum developers were 
instructed to ''think like mathematicians" and write down as many words 
as they could think of which are related to each of the 12 key concepts. 
They were allowed 1 minute per concept. 

For the card-sorting test, they were asked to sort the 12 concepts 
into piles. Similar concepts were to be placed within a pile. No 
restriction was put on the number of piles that could be formed. 

For the graph-building test, they were asked to construct a linear 
graph using the 12 concepts. They were instructed to select the 2 most 
similar concepts, connect them with a line, and label the line "1." 
Following additional rules, they continued the task until all 12 con- 
cepts were connected with 11 lines. 

With all three methods, proximity measures between the key concepts 
were formed. These proximities were examined with Johnson's (1>J}67) 
hierarchical clustering technique. 

Perhaps an example of one method of converting data from a measure 
of cognitive structure to proximities would be helpful. Consider the 
word-association data in Table 1. The> represent a hypothetical subjects' 



Insert Table 1 here. 



responses to two concepts in operational system. To the word set , he 
responds set (implicitly or explicitly), collection, element, finite. 
To the word element , he responds element (implicitly or explicitly), 
set, binary operation, finite. 
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A measure of the similarity between the concepts of set and element, 
for this subject, is the relatedness coefficient (RC) (Garskoff & Houston, 
1963) shown in Table 1. In the numerator, A • B represents the rank 
order of words under A which are shared in common with B and the rank 
order of words in B which are shared in common with A. In Table 1, the 
common words are set, element, and finite. In the denominator, A • B 
represents the 'rank order of words in A multiplied by the rank order of 
words in B. N* represents the rlumber of words in the longer of the two 
lists, and p represents some fixed numbei' greater than zero which weights 
various portions of the subject's response lists. We have set p=l which 
gives equal weight to all portions of the response lists in Table 1. 
The relatedness coefficient ranges from 0.0 to +1.00; the RC in Table 1 
for the concepts of set and element for our hypothetical subject is 
0.724. 

RESULTS 

Content Structure 

The largest entry in the distance matrix representing content 
structure was 5; this value occurred in one cell. These data, then, 
suggest a tight, formal structure. Since a hierarchical structure was 
expected, Johnson's (1967) procedure was used to examine the distance 
data. A meaningful hierarchical structure was retrieved (Figure 1). 



Insert Figure 1 here. 



At the apex is one cluster containing the 12 key concepts. From the 
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apex, two clusters emerged: (a) defining characteristics (DINARY OPERA- 
TION, ELEMENT. FINITE/INFINITE, ORDERED PAIR, SET), and (b) fundamental 
properties (ASSOCIATIVITY, COMMUTATIVITY, FUNDAMENTAL PROPERTIES, 
IDENTITY ELEMENT, INVERSES, the concept of OPERATIONAL SYSTEM, ROUND- 
NL'SS). Fn n the defining characteristics cluster, (a) sets (FINITE/ 
INFINTE, SET) were distinguished from (b) operations on elements (BINARY 
OPERATION, ELEMENT, ORDERED PAIR). From the fundamental properties 
cluster, (a) inverses (not in itself a fundamental property but required 
for the roundness property) and (b) fundamental properties were disting- 
uished. Lower levels make further distinctions which are consistent 
with our understanding of the subject matter. 

Cognitive Structure 

One might view prose materials as a measure of the way in which the 
authors interrelated key concepts in their memories; i.e., a measure of 
cognitive structure (Shavelson, 1972, in press). The digraph analysis 
of content structure, then, might be considered a measure of the authors' 
cognitive structure. If so, then alternative measures of the authors' 
cognitive structure should provide representations of cognitive struc- 
ture similar to the one found with the digraph analysis. To examine 
this hypothesis, data from the three measures of cognitive structure 
were compared with the data on content structure in Figure 1. 

Figures 2 and 3 present the results of the hierarchical cluster 
analysis of proximity data obtained from the word association and graph 
building methods. Since both curriculum developers sorted the 12 con- 
cepts into two piles--one pilsi contained the concept of operational 
system and the other pile contained all of the other concepts--cluster 
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analysis was unnecessary for these data. 

Insert Figures 2 & 3 here. 



A comparison of the results of the cluster analyses for the di- 
graph, word association, and graph building indicated that all three 
measurement methods produced similar structural representations of the 
concepts in operational systems. In all cases, the defining charac- 
t eristic s, of Operational Systems (Binary Operation, Element, Finite/ 
Infinite, Ordered Pair, and Set) were distinguished from the fundamental 
properties of Operational Systems [}\ssociativity , Commutativity , Identity 
Element, and Roundness (cf. Inverses]]. The one exception. Identity 
Element, was more closely related to defining characteristics than to 
fundamental properties by the word-association method. This can readily 
be explained: (a) Identity Element is a particular type of element; 
and/or (b) Identity Element provided an associative cue for relating 
this concept to Element. 

Additional distinctions among key concepts in the hierarchy were 
consistent with our understanding of operational systems and slight 
differences between hierarchies were interpretable. For example. 
Inverse is separated from fundamental properties in the content structure 
hierarchy but not in the cognitive structure hierarchies. Inverse is 
not, formally, a fundamental property of operational systems and this 
distinction was carefully maintained in the content structure. Inverse, 
however, is a necessary condition for the Roundness property; Roundness 
requires that an inverse exist for all pairs of elements in the opera- 
tional system. Thus, Inverse and Roundness cluster in data from the 
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measure of cognitive structure* 

The n^ethods presented here for examining content and cognitive 
structure and the data bearing on the validity of structural intorpreta- 
tions of these measures presented here and elsewhere (Gecslin, 1973; 
Shavelson, 1972, in press; Shavelson & Geeslin, 1973; Shavelson & 
Stanton, in preparation) indicate that meaningful representations of 
structure can l^e obtained. Such measure provide methods which might be 
used to evaluate the extent to which a mathematics curriculum communi- 
cates the structure it has been developed to communicate. They offer 
useful tools to the researcher concerned with how to communicate a 
subject-matter structure to a student effectively and efficiently. And, 
they may even provide diagnostic cools for adapting instructional 
methods to individual differences in students' knowledge of mathematics. 
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Table 1 

Data on a Word Association 
Test for a Hypothetical Subject 
Responding to Concepts in Operational Systems 



Response List 



Set 
set 

collection 

element 

finite 



rank 
4 
3 
2 
1 



Element rank 

element 4 

set 3 
binary operation 2 
finite 1 



Relatedness Coefficient (RC) = 



A • B 



(A • B) -[nP - (n - l)^' 



(4 2 1) U 



RC = 




Rr - 12+8+1 = 21 ^ Q 

TIFT 9 + 4 + 1) - 1 29 ^'^'^^ 
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Figure 1. 

Digraph Analysis of Content Structure: 
Results of the Hierarchical Cluster Analysis of Digraph 
Distances Between Key Concepts in Operational Systems 




Figure 2. 

Word Association Measure of Cognitive Structure: 
Results of the Hierarchical Cluster Analysis of Proximities 
(Relatedness Coefficients) Between Key Concepts in Operational Systems 
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Figure 3. 

Graph Construction Measure of Cognitive Structure: 
Results of the Hierarchical Cluster Analysis of Graph Distances 
Between Key Concepts in Operational Systems 



